Parkinson's Disease Detection - Project by Mrityunjay Abhijeet Bhanja¶
Parkinson’s Disease (PD) is a degenerative neurological disorder marked by decreased dopamine levels in the brain. It manifests itself through a deterioration of movement, including the presence of tremors and stiffness. There is commonly a marked effect on speech, including dysarthria (difficulty articulating sounds), hypophonia (lowered volume), and monotone (reduced pitch range). Additionally, cognitive impairments and changes in mood can occur, and risk of dementia is increased.
Traditional diagnosis of Parkinson’s Disease involves a clinician taking a neurological history of the patient and observing motor skills in various situations. Since there is no definitive laboratory test to diagnose PD, diagnosis is often difficult, particularly in the early stages when motor effects are not yet severe. Monitoring progression of the disease over time requires repeated clinic visits by the patient. An effective screening process, particularly one that doesn’t require a clinic visit, would be beneficial. Since PD patients exhibit characteristic vocal features, voice recordings are a useful and non-invasive tool for diagnosis. If machine learning algorithms could be applied to a voice recording dataset to accurately diagnosis PD, this would be an effective screening step prior to an appointment with a clinician
Independent Variable¶
ASCII subject name and recording number:
- name
Vocal Fundamental Frequency:
- MDVP:Fo(Hz) - Average
- MDVP:Fhi(Hz) - Maximum
- MDVP:Flo(Hz) - Minimum
Measures of variation in fundamental frequency:
- MDVP:Jitter(%)
- MDVP:Jitter(Abs)
- MDVP:RAP
- MDVP:PPQ
- Jitter:DDP
Measures of variation in amplitude:
- MDVP:Shimmer
- MDVP:Shimmer(dB)
- Shimmer:APQ3
- Shimmer:APQ5
- MDVP:APQ
- Shimmer:DDA
Two measures of ratio of noise to tonal components in the voice:
- NHR
- HNR
Two nonlinear dynamical complexity measures:
- RPDE
- D2
Signal fractal scaling exponent: DFA
Three nonlinear measures of fundamental frequency variation:
- spread1
- spread2
- PPE
Target Variable:¶
status - Health status of the subject (one) - Parkinson's, (zero) - healthy
In [24]:
#importing necessary libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from scipy import stats
from scipy.stats import pearsonr
from sklearn import metrics
from sklearn.preprocessing import StandardScaler, RobustScaler, MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, roc_curve, auc, classification_report, recall_score, precision_score, f1_score
from sklearn.model_selection import cross_val_score
import pandas_profiling
from xgboost import XGBClassifier
import warnings
warnings.filterwarnings("ignore")
In [25]:
#reading the data
df = pd.read_csv("Data - Parkinsons")
df.head()
Out[25]:
In [26]:
df.shape
Out[26]:
- The dataset has 24 columns and 195 rows.
In [27]:
df.dtypes
Out[27]:
In [28]:
df.astype({'status': 'category'}).dtypes
Out[28]:
- The target variable is "status", which is the indicator if a person has Parkinson's disease or not
- From all the independent variables, name is the only variable of type "object". Rest all are numeric in nature.
In [29]:
df.isnull().sum()
Out[29]:
In [30]:
df.isna().sum()
Out[30]:
There are no missing or null values.¶
In [31]:
df.describe().T
Out[31]:
The mean and median of most of the attributes seem to differ a lot by margin. This will determine the skewness of each column. We will take a look at each of them below.
All the values in spread1 are negative. We will scale them later for better model building.
Analysing output class¶
In [32]:
plt.figure(figsize=(5,5))
plt.style.use("ggplot")
plt.title("Output Class (status) Distribution")
plt.xticks([0,1])
sns.countplot(df["status"]);
plt.show();
n_true = len(df.loc[df["status"] == True])
n_false = len(df.loc[df["status"] == False])
print("Number of positive cases: {0} ({1:2.2f}%)".format(n_true, (n_true / (n_true + n_false)) * 100 ))
print("Number of negative cases: {0} ({1:2.2f}%)".format(n_false, (n_false / (n_true + n_false)) * 100))
We've got a 3:1 ratio of output values¶
In [33]:
for col in df.columns:
print(col)
The attribute "name" is irrelevant to our problem. The recording name will not provide any value to our model. It's only work is to act as a unique ID for all records, and hence, we'll be dropping the columns from our dataframe.¶
In [34]:
df = df.drop("name", axis=1)
In [35]:
df.head()
Out[35]:
In [36]:
sns.pairplot(df)
Out[36]:
Plotting all numerical attributes¶
In [37]:
#loop to plot all numerical attributes
for i, col in enumerate(df.drop("status",axis=1).columns):
plt.style.use('seaborn-pastel')
# create a subplot with 2 windows: one boxplot, one histogram
f, (ax_box, ax_hist) = plt.subplots(2, sharex=True, gridspec_kw={"height_ratios": (.15, .85)}, figsize=(12,6))
sns.boxplot(df[col], ax=ax_box, color="indianred");
sns.distplot(df[col], ax=ax_hist, color="indianred");
#vertical lines for median and mode
plt.axvline(np.median(df[col]),color='darkblue', linestyle='-.',label="Median", lw=3)
plt.axvline(np.mean(df[col]),color='green', linestyle=':',label="Mean", lw=3)
ax_box.set(xlabel='')
ax_hist.set(xlabel=col)
plt.legend(loc="lower right")
x=df[col]
mu = x.mean()
median = np.median(x)
sigma = x.std()
textstr = '\n'.join((r'$\mu=%.5f$' % (mu, ),
r'$\mathrm{median}=%.5f$' % (median, ),
r'$\sigma=%.5f$' % (sigma, )))
# place a text box in upper left in axes coords
props = dict(boxstyle='round', facecolor='wheat', alpha=1)
ax_hist.text(0.70, 0.95, textstr, transform=ax_hist.transAxes, fontsize=14,
verticalalignment='top', bbox=props)
plt.show();
#skewness
skewness = stats.skew(df[col]);
if(abs(skewness)<0.5):
print(f"Skewness of {col} is: {round(skewness,3)}, hence the distribution is fairly normal." )
elif(skewness>0.5):
print(f"Skewness of {col} is: {round(skewness,3)}, hence it is right skewed" )
else:
print(f"Skewness of {col} is: {round(skewness,3)}, hence it is left skewed" )
print("------------------------------------------------------------------------")
In [38]:
plt.figure(figsize=(30,20))
sns.heatmap(df.corr(), annot=True, linewidths=.1, fmt= '.1f', center = 1 ) # heatmap
plt.show()
It's evident that a lot of the independent variables are highly correlated. To check high correlations, we highlight them on the heatmap.¶
In [39]:
# High correlation visualization
# Absolute values only.
cutoff = 0.9 # only datapoints with correlations higher than or equal to cutoff to be plotted.
hi_corr_df = df.drop('status', axis=1).corr()[abs(df.corr())>=cutoff].round(2)
In [40]:
plt.figure(figsize=(30,20))
sns.heatmap(hi_corr_df.corr(), annot=True, linewidths=.1, fmt= '.1f', center = 1 ) # heatmap
plt.show()
The above highlighted variables have really high correlation (>0.9)
Bi-variate analysis¶
In [41]:
for i, col in enumerate(df.drop(labels=['status'],axis=1).columns):
plt.figure(i)
plt.xlabel('status');
plt.ylabel(col);
sns.swarmplot(y=col, data=df, x="status")
corr, _ = pearsonr(df['status'], df[col])
plt.title(f"Status v/s {col}", fontsize=15)
plt.show();
print(f"Correlation between Status and {col} is {round(corr,2)}")
print("___________________________________________________")
There isn't much significant relationship between individual independent variables v/s the target variable. Hence, we will all 22 columns for building our model.¶
In [42]:
X = df.drop(labels=['status'], axis=1) #dropping target variable
X.head(10)
Out[42]:
In [43]:
y = df["status"] #target variable in y
pd.DataFrame(y).head()
Out[43]:
In [44]:
#splitting in a 70:30 ratio
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
All our numeric attributes had different ranges hence they need to be put on a uniform scale for model building.¶
In [45]:
#scaling the values
scaler = MinMaxScaler(feature_range=(0,1)) #scale of 0-1
In [46]:
##min-max scaling
x_train = scaler.fit_transform(x_train)
x_train = pd.DataFrame(x_train, columns=X.columns)
x_train.head() #train values after scaling
Out[46]:
In [47]:
x_test = scaler.transform(x_test)
x_test = pd.DataFrame(x_test, columns=X.columns)
x_test.head() #test values after scaling
Out[47]:
In [48]:
from sklearn.linear_model import LogisticRegression
# Fit the model on training set
logistic = LogisticRegression(solver="liblinear")
logistic.fit(x_train, y_train)
Out[48]:
In [49]:
#predict on test
y_predict = logistic.predict(x_test)
y_predict ##our predicted values on test set
Out[49]:
In [50]:
logistic_score_train = logistic.score(x_train, y_train)
print(f"Train accuracy for logistic regression is {logistic_score_train}")
logistic_score_test = logistic.score(x_test, y_test)
print(f"Test accuracy for logistic regression is {logistic_score_test}")
Confusion Matrix for Logistic Regression¶
In [51]:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_predict, labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g');
In [52]:
LR_precision = precision_score(y_test, y_predict)
print(f"Logistic Regression - Precision: {LR_precision}")
LR_recall = recall_score(y_test, y_predict)
print(f"Logistic Regression - Recall: {LR_recall}")
LR_f1 = f1_score(y_test, y_predict)
print(f"Logistic Regression - F1 Score: {LR_f1}")
Classification Report¶
In [53]:
print(metrics.classification_report(y_test, y_predict, labels=[1, 0]))
ROC curve and area under the curve for logistic regression¶
In [54]:
LRprob=logistic.predict_proba(x_test)
fpr1, tpr1, thresholds1 = roc_curve(y_test, LRprob[:, 1])
roc_aucLR = auc(fpr1, tpr1)
print("Area under the ROC curve : %f" % roc_aucLR)
In [131]:
#ROC Curve for logistic regression
plt.figure(figsize = (12,6))
plt.style.use('seaborn-darkgrid')
plt.plot(fpr1,tpr1)
plt.plot([0,1],[0,1],'r--')
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC for Logistic Regression')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
In [56]:
LR = pd.DataFrame({'Accuracy': [logistic_score_test],
'Precision': [LR_precision],
'Recall': [LR_recall],
'F1 Score': [LR_f1],
'AUC':[roc_aucLR]},index=["Logistic Regression"])
In [57]:
LR
Out[57]:
2.Naive Bayes¶
In [58]:
from sklearn.naive_bayes import GaussianNB # using Gaussian algorithm from Naive Bayes
from sklearn.naive_bayes import BernoulliNB # using Bernoulli algorithm from Naive Bayes
# create the model
gauss = GaussianNB()
bern = BernoulliNB()
gauss.fit(x_train, y_train)
print(f"Test Accuracy for GaussianNB: {gauss.score(x_train, y_train)}")
bern.fit(x_train, y_train)
print(f"Test Accuracy for BernoulliNB: {bern.score(x_train, y_train)}")
In [59]:
y_pred_gauss = gauss.predict(x_test)
y_pred_bern = bern.predict(x_test)
test_score_NB = accuracy_score(y_test, y_pred_bern)
print(f"Test Accuracy for GaussianNB: {gauss.score(x_test, y_test)}")
print(f"Test Accuracy for BernoulliNB: {test_score_NB}")
Confusion Matrix for Naive Bayes¶
In [60]:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred_bern, labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g');
In [61]:
NB_precision = precision_score(y_test, y_pred_bern)
print(f"Naive Bayes - Precision: {NB_precision}")
NB_recall = recall_score(y_test, y_pred_bern)
print(f"Naive Bayes - Recall: {NB_recall}")
NB_f1 = f1_score(y_test, y_pred_bern)
print(f"Naive Bayes - F1 Score: {NB_f1}")
Classification Report¶
In [62]:
print(metrics.classification_report(y_test, y_pred_bern, labels=[1, 0]))
ROC curve and area under the curve for Naive Bayes (Bernoulli)¶
In [63]:
NBprob=bern.predict_proba(x_test)
fpr2, tpr2, thresholds2 = roc_curve(y_test, NBprob[:, 1])
roc_aucNB = auc(fpr2, tpr2)
print("Area under the ROC curve : %f" % roc_aucNB)
In [132]:
#ROC Curve for NB
plt.figure(figsize = (12,6))
plt.style.use('seaborn-darkgrid')
plt.plot(fpr2,tpr2)
plt.plot([0,1],[0,1],'r--')
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC for Naive Bayes')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
In [65]:
NB = pd.DataFrame({'Accuracy': [test_score_NB],
'Precision': [NB_precision],
'Recall': [NB_recall],
'F1 Score': [NB_f1],
'AUC':[roc_aucNB]},index=["Naive Bayes"])
In [66]:
NB
Out[66]:
3. kNN Classifier¶
In [67]:
# instantiate learning model (k = 3)
knn = KNeighborsClassifier(n_neighbors = 3)
# fitting the model
knn.fit(x_train, y_train)
# predict the response
y_pred = knn.predict(x_test)
# evaluate accuracy
print(f"Score for k=3: {accuracy_score(y_test, y_pred)}")
# instantiate learning model (k = 5)
knn = KNeighborsClassifier(n_neighbors=5)
# fitting the model
knn.fit(x_train, y_train)
# predict the response
y_pred = knn.predict(x_test)
# evaluate accuracy
print(f"Score for k=5: {accuracy_score(y_test, y_pred)}")
# instantiate learning model (k = 9)
knn = KNeighborsClassifier(n_neighbors=9)
# fitting the model
knn.fit(x_train, y_train)
# predict the response
y_pred = knn.predict(x_test)
# evaluate accuracy
print(f"Score for k=9: {accuracy_score(y_test, y_pred)}")
knn = KNeighborsClassifier(n_neighbors = 11)
# fitting the model
knn.fit(x_train, y_train)
# predict the response
y_pred = knn.predict(x_test)
# evaluate accuracy
print(f"Score for k=11: {accuracy_score(y_test, y_pred)}")
In [68]:
knn = KNeighborsClassifier(n_neighbors = 9)
# fitting the model
knn.fit(x_train, y_train)
train_score_knn = knn.score(x_train, y_train)
print(f"Train Score for KNN: {train_score_knn}")
# predict the response
y_pred = knn.predict(x_test)
test_score_knn = knn.score(x_test, y_test)
# evaluate test accuracy
print(f"Test Score for KNN: {test_score_knn}")
Accuracy of 94.9% of test set
Confusion Matrix¶
In [69]:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred, labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g');
In [70]:
knn_precision = precision_score(y_test, y_pred)
print(f"kNN - Precision: {knn_precision}")
knn_recall = recall_score(y_test, y_pred)
print(f"kNN - Recall: {knn_recall}")
knn_f1 = f1_score(y_test, y_pred)
print(f"kNN - F1 Score: {knn_f1}")
Classification Report¶
In [71]:
print(metrics.classification_report(y_test, y_pred, labels=[1, 0]))
In [72]:
# ROC curve and area under the curve for kNN
knnprob=knn.predict_proba(x_test)
fpr3, tpr3, thresholds4 = roc_curve(y_test, knnprob[:, 1])
roc_aucknn = auc(fpr3, tpr3)
print("Area under the ROC curve : %f" % roc_aucknn)
In [133]:
#ROC Curve for kNN
plt.figure(figsize = (12,6))
plt.style.use('seaborn-darkgrid')
plt.plot(fpr3,tpr3)
plt.plot([0,1],[0,1],'r--')
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC for kNN')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
In [74]:
knn = pd.DataFrame({'Accuracy': [test_score_knn],
'Precision': [knn_precision],
'Recall': [knn_recall],
'F1 Score': [knn_f1],
'AUC':[roc_aucknn]},index=["kNN"])
In [75]:
knn
Out[75]:
Ensemble Models: Random Forest, Bagging Classifier, XGBoost¶
4. Random Forest¶
In [76]:
from sklearn.ensemble import RandomForestClassifier
rfcl = RandomForestClassifier(n_estimators = 50, random_state=1,max_features=12)
rfcl = rfcl.fit(x_train, y_train)
In [77]:
y_pred = rfcl.predict(x_test)
test_Score_RF = rfcl.score(x_test, y_test)
print(f"Random Forest Test score: {test_Score_RF}")
Confusion Matrix¶
In [78]:
cm=confusion_matrix(y_test, y_pred,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g')
Out[78]:
In [79]:
RF_precision = precision_score(y_test, y_pred)
print(f"Random Forest - Precision: {RF_precision}")
RF_recall = recall_score(y_test, y_pred)
print(f"Random Forest - Recall: {RF_recall}")
RF_f1 = f1_score(y_test, y_pred)
print(f"Random Forest - F1 Score: {RF_f1}")
Classification Report¶
In [80]:
print(metrics.classification_report(y_test, y_pred, labels=[1, 0]))
In [81]:
# ROC curve and area under the curve for RandomForest
RFprob=rfcl.predict_proba(x_test)
fpr4, tpr4, thresholds4 = roc_curve(y_test, RFprob[:, 1])
roc_aucRF = auc(fpr4, tpr4)
print("Area under the ROC curve : %f" % roc_aucRF)
In [134]:
#ROC Curve for RandomForest
plt.figure(figsize = (12,6))
plt.style.use('seaborn-darkgrid')
plt.plot(fpr4,tpr4)
plt.plot([0,1],[0,1],'r--')
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC for Random Forest')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
In [83]:
RF = pd.DataFrame({'Accuracy': [test_Score_RF],
'Precision': [RF_precision],
'Recall': [RF_recall],
'F1 Score': [RF_f1],
'AUC':[roc_aucRF]},index=["Random Forest"])
In [84]:
RF
Out[84]:
5. Bagging Classifier¶
In [85]:
from sklearn.ensemble import BaggingClassifier
bgcl = BaggingClassifier(n_estimators=50,random_state=1)
bgcl = bgcl.fit(x_train, y_train)
In [86]:
y_pred = bgcl.predict(x_test)
BC_test_score = bgcl.score(x_test , y_test)
print(f"The score on test data for Bagging is {BC_test_score}")
Confusion Matrix¶
In [87]:
cm=confusion_matrix(y_test, y_pred,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g');
In [88]:
BC_precision = precision_score(y_test, y_pred)
print(f"Bagging Classifier - Precision: {BC_precision}")
BC_recall = recall_score(y_test, y_pred)
print(f"Bagging Classifier - Recall: {BC_recall}")
BC_f1 = f1_score(y_test, y_pred)
print(f"Bagging Classifier - F1 Score: {BC_f1}")
Classification Report¶
In [89]:
print(metrics.classification_report(y_test,y_pred,labels=[0,1]))
In [90]:
# ROC curve and area under the curve for BaggingClassifier
baggingcl=bgcl.predict_proba(x_test)
fpr5, tpr5, thresholds5 = roc_curve(y_test, baggingcl[:, 1])
roc_aucBC = auc(fpr5, tpr5)
print("Area under the ROC curve : %f" % roc_aucBC)
In [135]:
#ROC Curve for BaggingClassifier
plt.figure(figsize = (12,6))
plt.style.use('seaborn-darkgrid')
plt.plot(fpr5,tpr5)
plt.plot([0,1],[0,1],'r--')
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC for Bagging Classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
In [92]:
BC = pd.DataFrame({'Accuracy': [BC_test_score],
'Precision': [BC_precision],
'Recall': [BC_recall],
'F1 Score': [BC_f1],
'AUC':[roc_aucBC]},index=["Bagging Classifier"])
In [93]:
BC
Out[93]:
6. XGBoost¶
In [94]:
xgboost = XGBClassifier()
xgboost.fit(x_train, y_train)
Out[94]:
In [95]:
y_pred = xgboost.predict(x_test)
y_pred
Out[95]:
In [96]:
xgboost_score_test = xgboost.score(x_test, y_test)
print(f"Test score for xgboost is {xgboost_score_test}")
In [97]:
print(classification_report(y_test, y_pred))
Confusion Matrix¶
In [98]:
cm=confusion_matrix(y_test, y_pred,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g');
In [99]:
xgb_precision = precision_score(y_test, y_pred)
print(f"XGBoost - Precision: {xgb_precision}")
xgb_recall = recall_score(y_test, y_pred)
print(f"XGBoost - Recall: {xgb_recall}")
xgb_f1 = f1_score(y_test, y_pred)
print(f"XGBoost - F1 Score: {xgb_f1}")
Classification Report¶
In [100]:
print(metrics.classification_report(y_test,y_pred,labels=[0,1]))
In [101]:
# ROC curve and area under the curve for XGBoost
xgboostProb=xgboost.predict_proba(x_test)
fpr6, tpr6, thresholds6 = roc_curve(y_test, xgboostProb[:, 1])
roc_aucxgb = auc(fpr6, tpr6)
print("Area under the ROC curve : %f" % roc_aucxgb)
In [136]:
#ROC Curve for XGBoost
plt.figure(figsize = (12,6))
plt.style.use('seaborn-darkgrid')
plt.plot(fpr6,tpr6)
plt.plot([0,1],[0,1],'r--')
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC for XGBoost')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
In [103]:
xgb = pd.DataFrame({'Accuracy': [xgboost_score_test],
'Precision': [xgb_precision],
'Recall': [xgb_recall],
'F1 Score': [xgb_f1],
'AUC':[roc_aucxgb]},index=["XGBoost"])
In [104]:
xgb
Out[104]:
Meta-Classifier¶
7. Stacking¶
In [105]:
from mlxtend.classifier import StackingClassifier
In [106]:
clf1 = KNeighborsClassifier(n_neighbors=1)
clf2 = LogisticRegression() # RandomForestClassifier(random_state=1)
clf3 = BernoulliNB()
clf4 = RandomForestClassifier(random_state=1)
In [107]:
rf = RandomForestClassifier(random_state=1)
In [108]:
sclf = StackingClassifier(classifiers=[clf1, clf2, clf3, clf4], meta_classifier = rf)
In [109]:
for clf, label in zip([clf1, clf2, clf3, clf4, sclf], ['KNN', 'LR', 'Naive Bayes', 'RF',
'StackingClassifier']):
scores = cross_val_score(clf, x_train, y_train, cv=15, scoring='accuracy')
print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))
In [110]:
sclf.fit(x_train,y_train)
Out[110]:
In [111]:
y_pred = sclf.predict(x_test)
y_pred
Out[111]:
In [112]:
sclf_score_test = sclf.score(x_test,y_test)
print(f"Test score for stacking classifier is {sclf_score_test}")
In [113]:
print(classification_report(y_test, y_pred))
Confusion Matrix¶
In [114]:
cm=confusion_matrix(y_test, y_pred,labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["No","Yes"]],
columns = [i for i in ["No","Yes"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True ,fmt='g');
In [115]:
sclf_precision = precision_score(y_test, y_pred)
print(f"Stacking Classifier - Precision: {sclf_precision}")
sclf_recall = recall_score(y_test, y_pred)
print(f"Stacking Classifier - Recall: {sclf_recall}")
sclf_f1 = f1_score(y_test, y_pred)
print(f"Stacking Classifier - F1 Score: {sclf_f1}")
Classification Report¶
In [116]:
print(metrics.classification_report(y_test,y_pred,labels=[0,1]))
In [117]:
# ROC curve and area under the curve for StackingClassifier
stackProb=sclf.predict_proba(x_test)
fpr7, tpr7, thresholds7 = roc_curve(y_test, stackProb[:, 1])
roc_aucsc = auc(fpr7, tpr7)
print("Area under the ROC curve : %f" % roc_aucsc)
In [137]:
#ROC Curve for StackingClassifier
plt.figure(figsize = (12,6))
plt.style.use('seaborn-darkgrid')
plt.plot(fpr7,tpr7)
plt.plot([0,1],[0,1],'r--')
plt.xlim([0.0,1.0])
plt.ylim([0.0,1.0])
plt.title('ROC for Stacking Classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
In [138]:
sc = pd.DataFrame({'Accuracy': [sclf_score_test],
'Precision': [sclf_precision],
'Recall': [sclf_recall],
'F1 Score': [sclf_f1],
'AUC':[roc_aucsc]},index=["Stacking Classifier"])
In [139]:
sc
Out[139]:
In [140]:
models = [LR, NB, knn, RF, BC, xgb, sc]
model = pd.concat(models)
In [141]:
model_names = ['Logistic', 'NaiveBayes', 'kNN', 'RandomForest' , 'BaggingClassifier', 'XGBoost', 'StackingClassifier']
Comparing models¶
In [142]:
model
Out[142]:
It is evident that XGBoost has the highest accuracy and the maximum F1 Score. The plot to visualize the following data:¶
In [146]:
#plot to view above metrics
plt.figure(figsize= (17,15))
plt.style.use("ggplot")
plt.subplot(3,2,1)
plt.barh(model_names,model['Accuracy'], edgecolor = 'black', alpha = 0.7)
plt.xlabel('Accuracy Comparision')
plt.subplot(3,2,2)
plt.barh(model_names,model['Precision'], edgecolor = 'black', alpha = 0.7)
plt.xlabel('Precision Comparision')
plt.subplot(3,2,3)
plt.barh(model_names,model['Recall'], edgecolor = 'black', alpha = 0.7)
plt.xlabel('Recall Comparision')
plt.subplot(3,2,4)
plt.barh(model_names,model['F1 Score'], edgecolor = 'black', alpha = 0.7)
plt.xlabel('F1 Score Comparision')
plt.subplot(3,2,5)
plt.barh(model_names,model['AUC'], edgecolor = 'black', alpha = 0.7)
plt.xlabel('ROC-AUC Comparision')
plt.show()
XGBoost has the highest accuracy and F1 score(0.966, 0.977 respectively) followed by Stacking Classifier,kNN, Bagging Classifier and Random Forest with accuracy of 0.949153. Among the models that came 2nd after XGBoost, kNN with 9 neighbours has the highest F1 score of 0.968421. All the models have performed very well, and give us a very high accuracy of identifying if a person has Parkinson's disease based on their vocal features.¶
So for this problem, XGBoost is the perfect classification algorithm.¶
In [ ]: